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ABSTRACT 

The issue of the perceived succe'ss of human services 
program evaluations is the specific focus of this paper. Five areas 
that affect the potential for producing, effective program evaluation 
form a framework for discussing factors related. to perceived success . 
The areas include inappropriate Conceptualization of what an 
evaluation can accomplish, what to evaluate and how to evaluate. 
Inappropriate recognition of the limitations of existing program data 
and inappropriate training of most evaluators also affect the 
evaluation. There is little consensus about what constitutes a 
successful evaluation, Seyond the recognition that success is related 
to the utilization of the information generated through the 
evaluation process. Some consider success to be the extent to which 
information generated on the impact of the program affects policy 
decisions regarding the future of that program. Some view an 
evaluation as "successful" if the program is better articulated. 
Others expect the information' generated by the evaluation to lead to 
more effective management. In other words, perceived success ol 
program evaluations has a variety of meanings' for the same ai 
different audiences. (Author/CM) 
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There is a growing realization that , while the demand 
for program evaluations is expanding rapidly, completed 
evaluations are rarely perceived to be "successful". there 
are dangers inherent in this paradox which may undermine the 
validity of the entire evaluation process. 

The issue of the perceived success of program evaluations 
is the specific focus of this paper. ^ Five areas that affect 
the potential for producing effective program evaluation 
form a framework for discussing factors related to perceived 
success. 

What constitutes a successful evaluation? There is 
little consensus around this issue beyond the recognition 
that success is related to the utilization of th^inf ormat ion 
generated through the evaluation process. Some consider 
success to be the extent to which information gehel^ated on 
the impact of the program affects policy decisions regarding 
the future of that program. Some view an evaluation as 
"successful" if the program ±s better articxAated. Others 
expect the information generated by the* evaluation to lead 
to more effective management » etc . In other words, perceived 
success of program evaluations has a variety of meanings for 
the same and different audiences. 
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AREAS THAT AFFECT THE POTENTIAL FOR EFFECTIVE PROGRAM EVALUATIONS . 

Five areas that affect the potential for producing 
effective program evaluations are listed. These areas form 
a framework for examining and elabotating on the issue of 
perceived success as it relates to effective program 
evaluations . 

1: Inapproprate conceptualization of what a 
program evaluation can accomplish. 

2. Inappropriate conceptualization of what to 
evaluate. 

3. Inappropriate recognition of the limitations of 
existing program data for evaluation purposes. 

4. Inappropriate conceptualization of how to 
^ evaluate. 

5. Inappropriate training of most evaluators. 

The balance of this paper is devoted to a disucssion of 
factors within each of these five areas that affect perceived 
success of program evaluations. The paper limits its discussion 
to program evaluations, which are evaluations of human service 
programs. 

1. INAPPROPRIATE CONCEPTUALIZATION OF WHAT A PROGRAM EVALUATION 
CAN ACCOMPLISH . 
Program evaluations are not unidimensional . They cannot 
be viewed from the perspective of a single success criterion. 
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There are many reasons for undertaking evaluations , 
(Chelinsky 1978, Rossi 1979, Rutman 1980). Of these, three 
broad categories are generally apcepted. These are evaluations 
undertaken f or r 

a) accountability and atidit purposes where monitoring, 
verification and fund allocation considerations 
predominate , 

b) management and administrative purposes where 
evaluation is primarily a management tool and 

c) policy and planning purposes where decisions on 
expansion or curtailment of the programs are 
expected to follow the evaluation. 

Viewed from another perspective, there are various 
approaches to evaluation. They range on a continilum from 
performing process evaluations, which focus on operating 
programs and their actual implementation, to doing outcome 
evaluations which focus on the impact of the program.' Many - 
view outcome or impact information and its effect on policy 
and planning as THE legitimate role for all evaluations. 
They view success in terms of the accuracy of the outcome 
information generated and the extent to which this information 
affects decisions about the future of the program. 

This is an inappropriate conceptualization of what 
determines success. It ignores the significant impact of 
process evaluation^on how managers operate their programs. 
Often it places too little emphasis on the program operation, 
on effectiveness or efficiency issues, or on other criteria 
that could be legitimately considered a successful utilization 
of the information generated. 



In other words, there is often an inappropriate 
conceptualization of what a program evalmatioh can accomplish. 
Various reasons for conducting evaluations and different 
methods of conducting them can have a significant impact 
on tfhe potential uses (and> abuses) of the information 
generated. If program evaluations are not to be inappropriately 
tarnished with tjie perception of failure, this point must be 
understood, accepted and emphasized by evaluators and others 
involved in evaluations. 

2. INAPPROPRIATE CONCEPTUALIZATION OF WHAT TO EVALUATE 

There is a lack of concensus on the definition of terms 
such as needs, programs, services and activities. This leads 
to inappropriate conceptualization of what to evaluate. 
Confusion of terms confounds the issue of determining if a 
program evaluation is successful, since measuring the success 
of "something" assumes the "something" is appropriately defined 
and understood. 

Figure I provides a framework that outlines and defines 
major categories utilized in th^ conceptualization of what 
to evaluate in himan service organizations. It begins with 
the concepjt of need or goal. This basic concept often is 
defined inappropriately. For instance, many acknowledge the 
"need" for day-care, for protected housing for the disabled 
and for more recreation and social services. Yet from an 
evaluator's perspective these are NOT needs. They are program 
strategies. The need is not day-care but an enriched 
environment for the child, providing an opportunity for mothers 
to fulfill themselves through further^ education or work, etc. 
The need is not protected housing, but to optimize the potential 

. 4 . 
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FIGURE I 



A CONCEPTUAL FRAMEWORK FOR EVALUATING 
mm\ SERVICE ORGANIZATIONS 



MAJOR CATEGORIES 

UTILIZED IN CONCEPTUALIZING 

HUMAN SERVICE ORGANIZATIONS 



DEFINITIONS OF THE 
MAJOR CATEGORIES 



Overall 

NEED* 

(or goal) 



A general statement of optimal 
states of desired human conditions 
for a special segment or of all 
of mankind. 



OVERALL 

Agency or Organization 

OBJECTIVE 

(or goal) 



An operational statement of 
the rationale for the , 
existence of the organization 
that links an organized 
response to a perceived need 
or needs. 



Human Service 

PROGRAM 



A group of services and 
activities undertaken as an 
organized response to assist 
in meeting the objectives of an 
organization . 



PROGRAM OBJECTIVE 



A statement of a strategy and 
rationale for an organized 
response to meeting the 
objectives of the organization. 



SERVICE 



Specific identifiable 
organized procedures undertaken 
to assist in meeting a 
program objective. 



'ACTIVITY 



A specific identifiable discrete 
task. 
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COMPONENTS OF THE 
MAJOR CATEGORIES 

Basic Human neeids sometimes 
referred to as quality of 
life indicators . 



The overall need or needs to 
which the organization is 
responding 

-PLUS- 

A general indication of the i 
broad strategies the organization 
employs to meet this need. 

One or more services which 
logically cluster together as 

an organized response to meeting - , 

the organizations objectives. i 

J* 

The specific strategy of the 
program 

-PLUS- 

the overall need to which this 
program strategy is i^esponding, 

\ 

An organized cluster of individual 
activities that are logically 
associated with each other. 
Services are the major elements 
of a human service program. 



Specific, narrowly-defined tasks 
such as driving or completing 
forms . When combyied with other 
tasks that cluster together, 
these tasks f orm ja service. 



for the disabled to lead a normaJL life, etc. The need is 
not recreation but to optimize physical well-being, etc: 

If the need (as in these examples) is defined in terms 
of the program strategy, the success of 'the program becomes 
a "numbers game" of measuring how much more of that strategy 
is provided; more day-care, more housing units, more recreation. 
Logically most agree that these are inappropriate success 
criteria, yet little effort is expended in defining appropriatel 
the needs to which organizations and programs are responding. 

Figure I illustrates how organization and program 
objectives rationally link overall heeds to program strategies. 
Figure II is an example of how an organization can utilize' 
the evaluation framework. It shows how an organization can 
be conceptualized, using the first three categories outlined 
in Figure I. It illustrates how a process evaluation can. 
affect an organization's ability to reconceptualize its programs 
and their potential impact. As a resul^of this process the 
organization can generate new information that should provide 
significantly better indications of the actual effects (outcome) 
of their programs. 

To define needs as optimal states of desired himan 
conditions, and programs as specific strategies undertaken 
to help achieve these desired states, sounds relatively 
simple, where in fact these definitions are difficult to 
operationalize . Differentiating among programs, services 
and activities is not a straight forward procedure. It is 
an art form. 

- 6 - 
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FIGURE II ll 

AN EXAMPLE OF AN ORGANIZATION 
UTILIZINGl, THE EVALUATION FRAMEWORK 

To optimize the potential of heart-disease patients and 
their families to lead a normal life. 



To optimize the potential of heart-disease patients and 
their families to lead a normal life by providing opportuniti-es 
for both . to: a) learn more about their potential to normalize their 

lives and, 

b) share their concerns with others who have experienced 
similar traumas . 



1. Hospital/Pre-discharge Support Program 

Objective: to assist in reducing the emotional stress arising from 
a cardiac crisis by: 

(a) providing an opportunity fbr patient, spouse and 
family to communicate and respond to perceived 
problems related to their cardiac episode and recovery 

(b) providing reassurance of continued support through an 
explanation of Heart to Heart and other community 
resources available to them. 

2 . Family Series Prog'^^am 

Objective: to assist in the achievement of an optimum lifestyle for 
heart-disease patients and their families by: 

(a) providing information, education and support through 
a series of group sessions , 

(b) providing an opportunity to communicate and respond 
on a one-to-one basis with professionals. 

(c) providing an opportunity for participants in the 
series to share mutual concerns and experience. 

(d) encouraging utilization of appropriate community 
t resources . 



3, ^ CoT(ftnunit'y Liaison Progr&m /* . • 

ObjectiVe: to opitimize the potential for members of the medical, 

professional and community-at- large to respond positively 
_ to pripblems related to heart disease by: . ^ L_ 



(a) assisting in- the development of support programs for * 
heart attack patients aru^ their families. 

(b) creating an ,aW&rene3^ of the problem as pe^rceived by 
heart attack pa^fients and their families. , 

-(c) creating an awarene^'ss of the rple of th6 family iri the 
recovery of heart attack patients': 

(d) encouraging the ul^ilization of Heairt Foundation 
resources, expertise, educational ^nd health contacts 
in the community , ^^terature, films and speakers. 

(e) providin]g support and encouraging, the utilization of ^ 
appropriate coramunityv resources such as exercise 
programs counselling services, etc. 



^ ' 4. Alumni Program 



Objective: extend the opportunities £or\ocial support and personjal 

, well-being for particit)ants of thV^Family Series by: 

(a) v providing a forum for participanta tp meet on a regular 

basis to continue sharing experiences and friendships. 

(b) providing an opportunity for, participants to receive 
further support and up-dated information. 



\Z (c) encourage participants to volunteer in on-going 

ERIC 



, ^ activities that further t^e y?ork of Heart to Heart and 

ir- Ontario Heart Foundation. 
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For evaluators, appropriate definitions of needs, programs, 
services and act'ivities that are related to the organization 
being evaluated cannot be assxamed or left to program staff 
tg define without guidance. Figure III extends the profile 
of one of the programs referred to in Figure II. The 
specific services ^nd activities employed to meet the 
program objectives are displayed, as is the relationship 
between a program, service and activity. The program is 
shown to consist of a cluster of services, and the services 
to be made up of clusters of activities (many of which are 
not spelled out in. this example) . ' 

Evaluators often ignore or only superficially address 
the issues raised above . This is an important factor that 
later in the evaluation process seriously affects the 
perceived success of the evaluation. 

\ . < ..... 

3. INAPPROPRIATE RECOGNITION OF THE LIMITATIONS Ot 
EXISTING PROGRAM DATA F6R EVALUATION PURPOSES . 

^ Most evaluators, while giving lip service to carefully 
defining programs and objectives, build their evaluations upon 
existing conceptualizations and information presented to them 
by program managers. This is a serious error that affects 
the entire evaluation. 

Truisms such as "how can you evaluate when you don't 
Icnow what you're evaluating" are 'ignored as evaluators and 
program^ managers ^pew out reams of irrelevant data. For 
instance, per diem costs of residential care are collected 
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FIGURE III 



PROGRAM PROFILE OF AN ORGANIZATION 
UTILIZING THE EVALUATION FRAMEWORK 
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Program #2 : Family Series Program 

Objective: to assist in the achievement of an optimum lifestyle for heart disease patients by: 

(a) providing information, education and support through a series of group 

sessions, ^ • 

(i) provide forxm with informal atmosphere (circular seating) which 
encourages discussion, questions, sharing, expressing concerns, 

(ii) provide educational materials (pamphlets from OHF, books, reading lists). 

\ 

(iii) provide consultants who clearly present up to date information on various 
aspects of heart disease (clarify and tonfirm previously acquired 
information) . 

(iv) provide ample opportunity for questions in group. 

(v) acquaint participants with resources in the cotamunity. 

(b) providing an opportunity to communicate and respond on a one-to-one basis with 

professionals. ' 

(i) provide opportunity for par^:icipants to individually meet after 
presentation to discuss specific concerns with the leader and/ or 
consultant. - ' •■ v 

(ii) provide office and leader's home telephone numbers. 

(c) providing an opportunity for participants in the series to share mutual t_ 

concerns and experiences. 

(i) facilitate interaction between participants bv encouraging them to 

share their experiences and concerns with each other, by asking questions, 
asking for and giving feedback, using checklists, etc. 

(ii) provide opportunity to meet and talk with each other after the 
presentation. ^ , . 

(iii) provide telephone number's and addri^sses of all participants' in group. 

(d) encourage utilization of appropriate community resources. 

(i) encourage the utilization of community resources. 

(ii) provide specific materials and Information to individuals with 
specific concerns (e.g. stop smoking., resources in community)* 
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MAJOR 
SERVICES 
OF 
^THIS 
PROGRAM 
WITH 
MAJOR 
ACTIVITIES 
ASSOCIATED 
WITH 
EACH 
SERVICE' 



without knowing what really happens to the client during 
the course of that day. Costs per adoption are an&lyzed 
without reference to changes in the profiles of those being 
adopted. As previously mentioned, defining the '*need" as 
day -care makes "success" a nxombers game of providing more 
day-care units, while ignoring what happens to the child 
involved in the day -care facility. Units of service rather 
than the effect of the service on the client become the 
focus, 

Evaluators agree that it DOES matter if financial and 
program, data are appropriate. Yet, they rationalize that 
time, money or limited mandates do not permit them to 
appropriately respond to these critical issues. They -get 
on with the job of the "real" outcome evaluation with too 
little attention given to the' issues related to the 
limitations of existing program data. 

An illustration of inappropriate program conceptualization 
is outlined in Figure IV. This figure Illustrates how a 
volunteer bureau typically conceptualizes its major programs. ' 
Figure V shows the major program as developed through the 
co-ordinated effects of volunteer bureau staff, volunteers, 
and an outside evaluator. The logic and content are substantially 
altered in Figure V. There is a more appropriate definition of 
what volunteer bureaus actually do. For example, administration 
is defined as a support service rather than a major program. 
The role of public relations as a support service for all 
programs is clarified. Programs are redefined to more 
rigorougsly deliniate services and the needs to which each 
pro'gram is responding. 



FIGURE IV 

AN ILLUSTRATION OF INAPPROPRIATE 
PROGRAM CONCEPTUALIZATION 



ONTARIO ASSOCIATION OF VOLUNTEER BUREAUX/ CENTRES 
INFORMATION SYSTEM 



COMPONENT ONg - REFERRAL 

RELATED GOAL: foWase the number of active and well placed volunteers 
in tWs community 

ACTIVITIES: Registration; Referrals 
COMPONENT TVJO - PROMOTION 

RELATED GOAL: To create and maintain an environment in JJe^ommunity 
conducive to voluntary participation and action. 

ACTIVITIES: Talks; Special Events 
COMPONENT THREE - EDUCATION 

RELATED GOAL: To enhance the development and operation of effective 
voluntary programs in the community. 

ACTIVITIES: Consultation; Training 
COMPONENT FOUR - COMMUNITY RELATIONS 

RELATED GOAL' To foster effective relationships and support co- 
REUTED GOAL. co-operative action between agencies to 

meet community needs. 
ACTIVITIES: Liaison Meetings 



COMPONENT FIVF - ADMINISTRATION 

RELATED GOAL: To achieve the above goals of the Volunteer Bureaux/ 
RELATED GOAL, lo^a^^^ effective and efficient manner. - 



ACTIVITIES: Budget; Staffing 



FIGURE V 

AN ILLUSTRATION OF APPROPRIATELY CONCEPTUALIZED PROGRAr^S- 



OUTMITO ASSOCJATWU 0^ l/QLUMTEER BUREAU/CENTRES 

PROGRAM PROFILE ' 



MAJOR PROGRAM AREAS 



SERl/ICES TO 
lUVJVJVUALS ANP 
GROUPS WISHING _ 
TO VOLUNTEER ^ 



ryPICAL SERVICES kUV 
IN EACH AREA 



ACTIl/iriES "yPICAL S11VV0V.T SERl 



;2 Re^eAAa^ 61/ peA-4oitfl-£ ok 
tzlcphom IntzfLvlziv^ . 

1 3 ¥oHovo-ap [a.) ' Initial u)Zth 

votantzzK 
,[b) ilnal utlth 
voluntzzfi 
- (cj i'incil vclth 
aQzncy 



CU-UkVLUA] IC}^ Uh 
ACTIl/ITIES WHICH 

eucouhage effective! 
voluuteeh programs 
amp volunteer 
participation 



aszncy volantzzfU . 
¥ acXJUtatlng pKO iz^^^-LcHal 
dzvzlopmznt. 0^ mcina.gzK6 . 
T OLclULtoLtlnQ ttwa^eite^A oi 
potzntldt ion. MotantzzfU 
to pa.A.tA.clpa.tz. 
ToLCltltCLtlnQ commanZcatlon 
bztwzzn volantzzK 
co-on.dZn.cit.on.6 . 



COUSULTATJOU RE:- 
VOLUNTEER PROGRAMS 
ANV VOLUNTEERJSU 



Con6altA,ns u)lth agzncy ^t; 
VolaKtzzK pKogKam6. 
Con^altZng viZth ZndlvldacLU / 
gfLoap^ izi pa.AtZclpa.tZon * 
Zonhu-ttJing Kzt voluntztK^ 
voZth ipzcXal nzzdA. 



IMPROl/EMENT 
STATUS' OF 
VOLUNTEERS 



OF 



Lobbying utZth any tzvzt oi 
govzfinmznt and/oA. oKganZza- 
tlom> ioK bznzilt^UappoKti 
ioK \)otu.ntzVi6. 
Encpu-Kaglng community 
Kzcognltlon oi thz Malaz 
oi \)oLantzz/u, " 



SERVICES TO 

comuNiv/ 



ShaKlng Inioimatlon vslth thz 
^znzfiat community on Kzriiiztt 
Thz dzMzLopmznt oi pzMonat 
pfLoillzi {l.z. 6klLl bank) 
In oKdzK to Kztpond to agzncy 
KZ((u.Z6t6 ioK specialized 
skills. 

VlKzct szimIcz- any piogKam 
anagzd within thz fau/teau. 



m 
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The point is that the data collected around programs 
such as those outlined in Figure IV are inappropriate for 
evaluation purposes. Despite this, evaluators tend to use 
existing data as if they were an appropriate base upon which 
to build their evaluations. The result is that the information 
gener^ated by the evaluation is not effectively utilized, 
since it rarely reflects the contextual reality of the * 
program. 

if 

Evaluators have the professional responsibility to 
avoid the accusation "we know better than we do." To build 
upon existing program data produced by most human service 
organizations is to limit the effective utilization and 
longer-term perceived success of the evaluation processes. 



INAPPROPRIATE CONCEPTUALIZATION OF HOW TO EVALUATE. 



Evaluations should not be formulated on the basis of a 
standardized methodological format. A number of studies of 
evaluationa have confirmed an inverse relationship! the 
BETTER the methodology, the LESS the utilization. (Rossi'et al., 
1978:178) It is beyond the scope of this paper to elaborate 
on this relationship. As stated previously, success depends 
upon utilization of the information generated by the 
evaluation. Issues such as standards, frameworks and rigorous 
methodology must be responsive to the multi-dimensional aspects 
of the evaluation process. 

The role of the evaluator must> be understood. Evaluators 
are not technicians pumping out information for a client. 
They are applied scientists concerned with the rigor of the 
methodology. However, they must recognize that this rigor' 
should be examined and applied in the context of the potential 

- 13 - 
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utilization of the information generated by the evaL^aation 
process. 

"The nature of the program, the purposes of the 
evaluation and the proces.s of the evalxiation all affect 
methodological issues. These and other contextual issues 
must be considered by fiunders, program managers and 
evaluators if the perceived success of evaluations is to 
be enhanced. 

5. INAPPROPRIATE TRAINING OF MOST EVALUATORS. 

> 

Evaluators are considered social scientists. Most 
receive their training in basic social science rather than 
applied -methodology . This basic methodology conforms to the 
classic scientific experimental design of pre-test, post-test, 
experimental group - control group, witfe random assignment 
to both groups. Explanation and prediction are its under- 
lying goals. Rossi (1978) sums up the difference between 
basic and applied research stating: "basic research is 
discipline oriented, applied research is decision oriented." 
The point is that they ARE different. 

Success in basic research implies the generation of 
reliable and valid information obtained through recognized 
standards of ^ methodological rigor. Success in applied 
research (which provides the framework for program evaluations) 
implies utilization of the information generated by the 
evaluation process. As stated above, methodological rigor 
in the applied field does not insure a successful evaluation, 
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Yet many evaluators continue to shrug off the fact that little 
attention is given to their findings, disclaiming responsibility 
-since they "adhered to a methodologically sound evaluation." 

In the long run this inappropriate perspective would be 
^f diminishing importance if new developments in applied 
methodology were being integrated into evaluation precedures. 
However, even in many new courses developed specifically to 
train evaluators, basic social scientists continue to 
perpetuate the myth that basic and applied methodology are 
almost identical. Students are instructed that the focus 
on rigorous methodology and outcomes is^ compatible and almost 
synonymous with utilization issues. Other issues, such as 
appropriately defining needs and program responses to those 
needs (which are basic to good evaluations), are minimized. 

This discussion is not meant to demean basic scientific 
methodology. Every good evaluator MUST have a thorough 
working knowledge of it. However, every good evaluato'r 
should also be exposed to a thorough working knowledge 
of the new* developments in applied methodology. The 
reality is that many now teaching courses in evaluation 
"don't even know that they don't know." \ * '^-^ 

Applied social scientists with considerable field 
experience should have a more active role in organizing 
courses and field placements, as well as training new 
applied social scientists, if we are to avoid one of the 
underlying causes for the perceived failure of many 
evaluations . 
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This paper has referred to five areas that influence 
the potential for effective program evaluations. The intent 
has been to draw attention to problem areas that affect the 
perceived success of the evaluations. The paper has 
suggested that if evaluations are to be* perceived as a 
success, evaluators should play a more active role in 
shaping the expectations of program evaluations, as well 
as'* insuring that issues arising out of these expectations 
are more appropriately met. 
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